[docs] Plan progressive tool disclosure for the playground build kit#5405
[docs] Plan progressive tool disclosure for the playground build kit#5405ashrafchowdury wants to merge 1 commit into
Conversation
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR adds planning documentation for progressive tool disclosure in the playground build kit. It defines the runner advertisement seam, two discovery meta-tools, execution and permission invariants, staged rollout, measurement, schema-diet work, scope boundaries, and implementation questions. No implementation changes are included. ChangesProgressive tool disclosure
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1cd91cd3-59eb-49de-9205-786764ff4f1d
📒 Files selected for processing (6)
docs/design/agent-workflows/projects/progressive-tool-disclosure/README.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/context.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/design.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/plan.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/research.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/status.md
| 5. Cost of laziness is bounded: ≤1 extra round-trip per distinct op used; a schema fetched at | ||
| most once per op per conversation. |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Define and schedule the required schema cache.
The context makes “at most once per op per conversation” a success criterion, but the design only calls caching optional and the plan assigns no implementation, ownership, lifecycle, or test.
docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md#L74-L75: retain the bound as an explicit acceptance criterion.docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md#L97-L101: specify cache ownership, invalidation, and cold-replay behavior.
📍 Affects 2 files
docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md#L74-L75(this comment)docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md#L97-L101
| - **`agenta_ops(query?)`** — returns the op catalog as a compact list: `{op, one_line, | ||
| read_only}` per op, no input schemas. Optional `query` filters. A few hundred tokens, flat, | ||
| regardless of catalog size. This is the "what can I do on the platform" index. |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Bound the agenta_ops result; it cannot be flat with catalog size.
Returning one entry per operation is O(n), so the result grows as the catalog grows. Define pagination, a hard limit, or bounded query semantics, and change “flat regardless of catalog size” to describe only the advertised prompt cost.
| A platform op is a `callback`-kind spec with a direct `call`; so is a `reference` (workflow) | ||
| tool. There is no explicit marker today (research seam 5). Two options: | ||
|
|
||
| - **Heuristic (zero wire change).** Collapse every direct-`call` callback spec into the | ||
| meta-toolset; leave builtins, `client`, `code`, `gateway` (callRef), and MCP advertised as-is. | ||
| The playground overlay only injects platform ops (+ the two client tools), so this covers the | ||
| measured cost. Risk: an author who added a `reference` tool would see it disclosed too. Fine for |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
The eligibility contract does not cover all platform operation specs.
The resolver emits handler-mode platform operations with call_ref and endpoint-mode operations with direct call; a direct-call-only heuristic therefore leaves part of the platform-op set advertised and can fail the planned advertisement test.
docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md#L54-L60: include both representations or require the platform source marker.docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md#L95-L100: correct the platform eligibility description.docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md#L36-L38: revise the POC recommendation or narrow its scope.docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md#L40-L50: align Slice 2 behavior and tests with the chosen eligibility contract.
📍 Affects 4 files
docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md#L54-L60(this comment)docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md#L95-L100docs/design/agent-workflows/projects/progressive-tool-disclosure/status.md#L36-L38docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md#L40-L50
| A playground author opens an agent and types a message. The model sees two small platform tools | ||
| (`agenta_ops` to list what it can do, `agenta_op` to fetch a schema and act) instead of a wall | ||
| of op schemas. A no-op turn costs a low-hundreds token constant instead of ~15K, and stays flat | ||
| as the catalog grows. Every op still runs with its exact self-targeting binding and approval | ||
| gate. Builds are cheaper and, per the internal-tools review, more reliable. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
State approval parity as a requirement, not an achieved guarantee.
The README says every operation still has its exact approval gate, but design.md and status.md identify target-spec permission gating as an unresolved implementation requirement. Reword this as an acceptance criterion until the per-operation approval tests pass.
1 participant
Context
Open a playground agent that shows "Tools: None", type "hi", and the turn costs about 15K
prompt tokens. The cost is the build kit: it advertises roughly 13 platform-op tool schemas to
the model on every turn, before the model has done anything. Two of those schemas
(
commit_revisionandtest_run) each embed the full ~5.5K-token agent-template schema, whichis about 11K of the 15K on its own. The same always-on tools also hurt reliability; the
internal-tools review found that extra visible tools become "wander targets" that derail runs.
Skills already avoid this. They live on disk, put only their name and description in the
prompt, and load the body on demand. The platform ops never got the same treatment.
What this adds
A planning workspace only. No code and no runtime change. It captures the problem, the current
wiring (grounded in file:line), a proposed design, and a sliced plan, so the team can agree on
the approach before anyone writes code.
The proposed approach: advertise a small discovery meta-toolset instead of the full op schemas,
and load a schema only when the model asks for it.
Before (every turn):
13 platform ops, each a full {name, description, inputSchema} in the prompt. ~15K tokens.
After (the design):
2 tools in the prompt.
agenta_ops()lists what the agent can do (op name + one line, noschemas).
agenta_op(op, args)fetches one op's schema on demand, then runs it. A no-op turnpays a low-hundreds constant and stays flat as the catalog grows.
The op specs stay resolved and private in the runner's memory, so execution does not change.
The invoker feeds the target op's private call descriptor into the same direct-call path, which
keeps the self-targeting $ctx bindings, the SSRF guard, and the per-op approval gate exactly as
they are today.
Files, under docs/design/agent-workflows/projects/progressive-tool-disclosure/:
For reviewers
The point of a plan-only PR is to settle these before code. Full detail in status.md:
agenta_opwith a describe/execute mode, or two tools? (rec: one)agenta_opon the target op's private spec, not on the invoker, so writeskeep their approval prompt. (rec: yes)
(rec: runner-side)
(rec: heuristic for the flagged POC, marker before default-on)
Notes
Docs only; nothing to run. The token numbers come from the 2026-07-17 investigation and are
flagged for a re-baseline in the first slice. The design is verified against the current runner
code; the seam is advertisedToolSpecs() in services/runner/src/tools/public-spec.ts, consumed at
pi-assets.ts and environment.ts.